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ABSTRACT 

A standardized exam for program-level assessment can take the form of 1) a customized exam 
developed in-house by faculty and linked explicitly to program-level learning goals; or 2) a 
standardized exam developed externally by assessment experts and linked to a set of somewhat 
broader and more generalizable learning goals. This article discusses the design, development 
and implementation of a customized exam and subsequent transition to a commercial exam (i. e ., 
the ETS Major Field Test), within an undergraduate business program. We discuss the lessons 
learned from our experience with the customized exam, our analysis of both the assessment 
process and the results gathered (primarily curriculum-related), and the rationale underlying the 
eventual migration to the commercial exam. Of particular emphasis is the situation-dependent 
and potentially complementary roles of the customized and commercial exams. In this regard we 
provide a comparison of the tw’o approaches through a framework based on a set of 
administrative and assessment considerations. They include: relevance to learning goals, exam 
design and development process, delivery of the exam, impact on learning, impact on courses and 
curriculum, and impact on student monitoring and management. We note that although the 
customized exam no longer exists as a standalone assessment instrument, it continues to play a 
role in assessment by complementing other methods. This outcome, as well as the process leading 
to it, are potentially applicable to other institutions pursuing an evolutionary approach to 
learning outcomes assessment. 

Keywords: Program-Level Outcomes Assessment; Standardized Exam; Major Field Test 


INTRODUCTION 



xternal stakeholders - particularly the federal and state governments, accrediting agencies, and potential 
students and their parents - are holding institutions of higher education increasingly accountable for the 
quality of education they are providing their students (Altbach, Gumport, & Berdahl, 2011). In the 
specific domain of business higher education, schools accredited by the Association to Advance Collegiate Schools 
of Business (AACSB) are subject to specific assessment standards, guidelines and reporting practices prescribed by 
the association (AACSB, 2012). In order to ensure and demonstrate student learning, business schools have 
responded by developing and implementing formal outcomes assessment plans (Bisoux, 2008). But while clear 
assessment planning and sound direct measurement is a critical prerequisite for program review, assurance of 
learning, and maintenance of accreditation, many schools have struggled with finding and implementing a 
sustainable process for outcomes assessment (Kuh & Ewell). 


Common approaches to outcomes assessment range from direct course-level assessment to direct and 
indirect program-level assessment (Marshall, 2007; Walvoord, 2004). Course-level assessment focuses on the 
operational elements of classroom learning and the specific learning goals of the course. It includes the use of pre- 
and post-testing, common exams, and grading rubrics, among others (Walvoord, 2004). By contrast, program-level 
assessment is a strategic, top-down approach consolidating assessment in individual courses, while also including 
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strategies that assess learning across the curriculum and over time (Marshall, 2007; Walvoord, 2004). Program- 
level assessment naturally attends to program-level goals, which in our organization address general business skills 
expected of all business students. 

In this paper we provide a description and analysis of ‘lessons learned’ from our use of multi-disciplinary 
proficiency exams to assess undergraduate student learning prior to graduation. Specifically we discuss the creation 
and delivery of two objective exams intended to assess learning in various business disciplines. The first exam, 
which essentially was experimental, was developed in-house and designed for implementation following the 
sophomore year. As such, it measured learning in each of the freshman and sophomore business core classes. After 
deployment for three consecutive years, the customized exam, which we had labeled the ‘Core Competency and 
Proficiency Exam’ (CCPE), was discontinued and replaced by the senior-level ETS Major Field Test, a standardized 
exam used by hundreds of business schools across the country. 

Our experience in deploying these exams over a 6-7 year period led to a number of observations and 
conclusions that guided both refinement of our assessment strategy as well as changes to our curriculum and 
delivery methods. As we discuss in more detail later, both of the exams provided valuable information regarding 
student learning outcomes. However, the process of creative experimentation that led to on-going changes in the 
content and delivery of the exams, particularly changes to the CCPE, provided significant insights into the nature 
and challenges of customized exam construction and exam delivery, as well as into the characteristics of our 
sophomore class. This in turn reveals insights that were gleaned from an analysis of two essentially similar but still 
very different exams. Both exams are similar in that they are objective tests seeking to assess learning in specific 
business disciplines at a particular point in time. But they are different in that they were developed by different 
groups of academics, targeted to different audiences, and delivered at different points in time. Viewed in this 
context, a comparison of the two exams over time provides highly useful and actionable feedback regarding the 
process of program-level outcomes assessment through testing. 

The remainder of the paper is as follows. A short literature review provides background regarding 
assessment of learning and, more specifically, the various uses of standardized, customized and commercial exams 
for outcomes assessment. Next we describe in some detail the conception, development and implementation of the 
CCPE, including the trial-and-error aspects of it and the statistical analysis of the results. We also describe how and 
why we eventually chose to discontinue the use of the CCPE and search for another method. Then we discuss, in 
somewhat less detail, the move to the ETS Major Field Test and the delivery strategy employed. We conclude by 
comparing and contrasting the two exams with regard to their relative advantages and disadvantages. 

EXAM-BASED APPROACHES TO ASSESSMENT OF LEARNING 

Assessment of student learning is widely considered to require a variety of direct and indirect measurement 
strategies (Mirchandani, Lynch, & Hamilton, 2001). A multi-modal approach often is required because each method 
in isolation is insufficient to fully assess student learning, (Michlitsch & Sidle, 2002). As such, institutions tend to 
employ a variety of methods (Pringle & Michel, 2007), and they naturally tend to focus on the methods that are 
considered most effective (Michlitsch & Sidle, 2002), such as portfolio analysis, case study evaluations, analysis of 
group assignments, and standardized exams (Peterson & Einarson, 2001; Ruhland & Brewer, 2001; Walstad, 2001; 
Walvoord, 2004). 

Within the context of a multi-modal approach, standardized exams are a common form of both program- 
and course-level assessment (Black & Duhon, 2010). They offer the advantages of convenience, the ability to assess 
an extensive range of topics, objective grading resulting in consistent evaluation across students and evaluators and 
finally offer the advantage of evaluating large numbers on students in a reasonably short time-frame (Callahan, 
Strandholm, & Dziekana, 2009). Weaknesses of standardized exams include the risk of ‘teaching to the test’ 
(Firestone, Monfils, & Schorr, 2004), as well as a strong bias toward subject-matter content at the expense of other 
types of learning such as communication (Barnett, Dascher, & Nicholson, 2004). 

Standardized exams can take one of two forms: they can either be developed in-house by the faculty of the 
assessing institution (i.e., customized), or they can be purchased from a commercial testing service (Callahan et ah. 
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2009). Each has its advantages and disadvantages. For customized exams, the major advantage is the ability to 
target the exam to the individual mission program goals of the department or school, thus ensuring that the unique 
characteristics of the program are addressed and highlighted (Angelo & Cross, 1993; Apostolou, 1999; Black & 
Duhon, 2010; Callahan et al., 2009). For these reasons, any results are more likely to be accepted by the faculty 
(Callahan et al., 2009). For commercial exams, the major advantages are a rigorous development process conducted 
by testing experts, as well as a common set of questions and topics that can be used to benchmark the program 
against similar programs at other schools (Black & Duhon, 2010; Callahan et al„ 2009; Mirchandani et al., 2001). 
The respective disadvantages in turn tend to mirror each other. A customized exam is somewhat idiosyncratic and 
therefore difficult to benchmark against other schools, and is developed by faculty who are not necessarily experts in 
test design. A commercial exam in turn is not directly linked to the mission or program goals (Allen, 2004; 
Apostolou, 1999), and as such the faculty might consider the results irrelevant (Callahan et al., 2009). 

Because of their advantages, and despite their drawbacks, commercial exams are used extensively for 
outcomes assessment. A widely used commercial exam is the Educational Testing Service’s Major Field Test 
(MFT), which from September 2010 to June 2011 was administered by over 400 colleges and universities (ETS, 
2012). As such, numerous studies have explored implementation of the MFT from a variety of perspectives, 
including: 

• the determinants of students performance on the MFT (see, for example, (Bagamery, Fasik, & Nixon, 
2005; Barboza & Pesek, 2012; Bush, Duncan, Sexton, & West, 2008; Bycio & Allen, 2007; Contreras, 
Badua, Chen, & Adrian, 2011; Mirchandani et al., 2001; Terry, Mills, Rosa, & Sollosy, 2009; Terry, Mills, 
& Sollosy, 2008; Thornton & Arbogast, 2012) 

• the MFT as an indicator of student engagement (Ward, Yates, & Soong, 2009, 2010, 2012) 

• the MFT as a tool for improving learning (Black & Duhon, 2010; Wilson, 2008) 

• a case or field study discussing specific implementations of the MFT (Bush et al., 2008; Contreras et al., 

2011) 

By contrast, relatively less attention and research has been devoted to the effectiveness of customized 
exams. A notable exception is (Callahan et al., 2009), who describe the process used to develop, validate and 
deploy a customized exam in-house, as well as the challenges inherent in such a process (noted above). Despite the 
challenges, the authors note that their effort thus far has been successful, and that they intend to continue to refine 
and enhance their customized instrument. Our work is complementary to the work of Callahan et al in that while we 
also began with the development and implementation of a customized exam, we ultimately chose to move on to a 
commercial approach. In the following sections, we discuss the evolution of our customized approach, the lessons 
learned from its implementation, and the factors that led us to replace it with the ETS Major Field Test. 

DESIGN, DEVELOPMENT AND DELIVERY OF THE CUSTOMIZED EXAM 

The evolution of the customized exam from inception to the first implementation took place over a six- 
month period. The exam was administered six semesters over a three-year period during which the logistics of 
administration were evaluated and changed and student assessment was reviewed. The three-year process included 
the following phases: 

1. Design and development of a customized assessment instrument (CCPE) 

2. Implementation/delivery of the customized instrument, through multiple iterations 

3. Review and analysis of the assessment data 

4. Review and evaluation of the customized approach, including lessons learned 

Phase 1: Initial Design of the Customized Exam (CCPE) 

The customized exam was designed to assess the nine freshman and sophomore core courses in the 
business school. These courses included (with abbreviations): Introduction to Information Systems (IS 7), Business 
Information Systems (IS 2), Business Law (BLaw), Financial Accounting (Fin Acct ), Managerial Accounting (Mgrl 
Acct ), Micro-Economics (Micro Econ ), Macro-Economics (Macro Econ), and Business Statistics I and II (BStats I 
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& II). The departments involved were asked to submit ten multiple choice questions for each of the ten learning 
objectives of each core course. This would create a pool of 100 questions for each of the nine courses to be 
assessed. The summer before the first administration of the exam was spent collecting the questions and inputting 
them in our Blackboard Course Management site. From this pool of test questions we created a 90 question 
assessment that represented the ten learning objectives from each of the nine core courses. 

Phase 2: Administration of the Assessment 

We initially chose to administer the exam in a junior level required course, believing that this would 
naturally enforce 100% participation. The exam was administered using an electronic bubble scoring sheet allowing 
for quick grading and printouts of detailed item analyses. From this analysis, each department would be able to 
receive the overall assessment grade, grades from each course sub-test and percent of students scoring on each of the 
ten questions for each of the core courses. The exam was given in the same junior level course during the fall and 
spring semesters. 

Phase 3: Review and Analysis of the Data 

At the end of each academic year, the scores for the Fall and Spring administrations were combined and 
summary statistics were given to each department responsible for teaching one or more of the nine courses assessed. 
The summary statistics included the distribution of the overall assessment grades, the grades for each of the core 
course subtests of ten questions each, the item analysis summarizing the percent of students correctly answering 
each question within the department’s subtests, and the corresponding point bi-serial analysis for each question. 
Student performance on the overall exam and each course subtest was externally validated with the student’s grade 
point average and grades received in each core course, respectively, using Pearson’s correlation coefficients. After 
administering the exam for at least two years, multiple linear regression was performed to identify the strongest 
predictors of how well a student would perform on this internally generated assessment. Independent predictors 
included the student’s GPA and the year the exam was administered. 

As part of the review of the results, we asked each department to reconsider each question selected for the 
assessment, and to rank the question for level of difficulty (easy, medium or hard) and for its category in Bloom's 
Taxonomy (Knowledge, Concept, Application, Analysis and Synthesis). Each department was given a detailed 
summary report which included: 

A. Overall Exam Performance 

1. The mean score for all 90 questions 

2. The correlation coefficient measuring association between student score and student GPA 

3. The results of the overall multiple linear regression analysis (following the second and third year) 

B. Individual Department Sub-tests Performances 

1. The mean subtest score for the ten questions assessing their particular core course(s) 

2. The correlation coefficient measuring association between student score and student grade for each 
core course assessed from their department 

3. The distribution of Bloom's taxonomy for each of the ten subtest questions for the core course(s) 

4. The distribution of the perceived level of difficulty for each of the ten subtest questions for the core 
course(s) 

5. The point bi-serial estimate for each of the ten subtest questions for the core course(s) 

Results: Program Review 

The overall program review was summarized using the correlation between the exam scores and student 
grade point average. Despite a low mean score of around 50% on the exam, the correlation between the exam scores 
and student grade point average was 0.514, indicating a strong external validation of the exam. While students 
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might be retaining only about 50% of the course material, the exam correctly identified higher grade point averages 
with higher CCPE performance. After three years of administering the exam, regression analysis confirmed that 
overall mean scores significantly improved annually and students with higher QPA’s scored significantly higher. 
The multiple linear regression model predicted an increase of approximately 2% with each subsequent year and 
about a 12.3% increase for every 1 point increase in QPA. 

The review also exposed a problem in student management, subsequently resolved, which became apparent 
prior to the administration of the first assessment. In the first year we discovered that many of our students were not 
‘on track’ within the proposed curriculum. Of the 221 eligible juniors, only 121 (or 54.6%) had successfully 
completed all nine of the freshman/sophomore core courses on time. Although this was problematic with regard to 
the administration and analysis of the exam, it also provided the unexpected benefit of highlighting a significant 
problem in student management. Thus, from an assessment perspective, we formulated a program-level response 
plan that addressed student class scheduling and enforced the suggested curriculum sequence. We also turned to the 
departmental summaries (detailed in the next section) to troubleshoot curricular changes in an effort to improve 
overall retention of core business outcomes. 


Results: Departmental Review 


The second set of results pertains specifically to student performance on the individual department sub¬ 
tests. Table 1 summarizes the mean scores (percentage of the ten questions correct) for each of the nine subtests, the 
mean course grade (on a 4-point grading scale) received by students for each of the nine core courses, and the 
correlation of student performance on the CCPE exam with grades earned for each of the nine courses. The table 
also includes the departmental question reviews regarding: 1) which percentage of questions within each subtest 
were internally rated as ‘written clearly’, 2) the percentage perceived as medium or highly difficult, and 3) the 
percentage labeled cognitively higher on Bloom’s Taxonomy (i.e.. Application, Analysis or Synthesis). Results are 
reported for each of the three years except the specialized item analysis which was only reported for the first year to 
be used to inform immediate curricular changes. 


Table 1: Mean Score, GPA, Score-Grade Correlation, and Question Review by Course 



IS 1 

IS 2 

Micro 

Econ 

Mgrl 

Econ 

Mgmt 

Acct 

Fin 

Acct 

BStats I 

BStats 2 

BLaw 

Mean Score % 










Yearl 

69.8 

54.0 

48.6 

40.7 

61.7 

51.0 

48.5 

41.5 


Year2 

47.0 

44.7 

51.8 

50.3 

40.8 

44.7 

47.2 

42.9 


Year3 

75.7 

75.7 

55.9 

55.4 

63.1 

44.7 

57.3 

43.2 

46.6 

Mean Course Grade 










Yearl 

3.00 

3.22 

2.70 

2.75 

2.98 

2.91 

3.11 

3.02 


Year2 

3.24 

3.22 

2.94 

3.07 

2.95 

2.79 

3.10 

2.98 


Year3 

2.76 

2.76 

2.71 

2.88 

2.95 

2.80 

2.54 

2.84 

2.77 

Correlation 










Yearl 

0.177 

0.23 

0.317 

0.507 

0.299 

0.202 

0.250 

0.253 


Year2 

0.266 

0.29 

0.405 

0.405 

0.207 

0.405 

0.335 

0.286 


Year3 

0.205 

0.033 

0.351 

0.086 

0.267 

0.405 

0.271 

0.139 

0.205 

Question review * 










% written clearly 



85.0% 

95.0% 

100% 

95.0% 

100% 

95.0% 


% medium/highly difficult 



100% 

90.0% 

60.0% 

80.0% 

75.0% 

75.0% 


% Bloom’s: Application, 



85.0% 

65.0% 

45.0% 

75.0% 

45.0% 

90.0% 


Analysis, Synthesis 











Data taken from Year 1 Only 


Phase 4: Informed Changes within Departments 

Analysis of the exam results informed and motivated a closer examination of the exam itself as well as the 
individual core courses. In the information systems department, for example, the results prompted significant 
changes in the learning objectives and syllabus for the first core IS course. Reporting the point bi-serial estimated 
for each question motivated faculty to conduct an in-depth review of each question that was on the exam, including 
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why it was included and whether it addressed any of the key learning objectives for the course. The course was 
completely revised based on key learning objectives and new assessment questions were submitted. Although the 
correlation between CCPE performance and course grades did not change greatly, there was a statistically 
significant improvement in the grades for the IS subtests. 

The results from the economics subtests were somewhat mixed. The Microeconomics subtest scores were 
among the lowest, while the Macroeconomics scores were among the highest. Both subtests scores correlated well 
with grades, and statistically significant improvements in mean subtest scores were observed from year 1 to year 3. 
The distribution of clearly written questions, medium to difficult questions, and higher cognitive questions was 
overall higher for both economics subtests. The in-depth review of these two courses resulted in a revision of the 
key learning goals and subsequent pool of assessment questions. 

A closer look at the Business Statistics sequence resulted in increased discussions among those teaching the 
sequence as well as those upper level courses that use statistics as a pre-requisite. Business Statistics I was 
streamlined to increase technology and cover more applied skills and communication of results, while more projects 
and case studies were infused in Business Statistics II. Interestingly, key learning objectives did not change but 
pedagogy changes were encouraged. Additionally, the department created a statistics retention exam to be 
administered in a core junior-level class to measure the effect of these pedagogic changes. 

Phase 5: Overall Review and Evaluation of the Customized Approach 

As noted, only 55% percent of the junior-level students had completed all nine of the freshman and 
sophomore core courses. The reasons for this were varied. Approximately 15% of the junior class is comprised of 
transfer students who came into the program with varied backgrounds. The remaining 85% includes students who 
failed one or more of the core courses, those who elected to avoid some of the courses they perceived as difficult, 
and those who participated in our study abroad program. We were faced with two options for responding to the 
problem. Option one was to assess only the students who had completed all nine of the core courses. This however 
would present a serious tracking problem and would result in assessing some students in their senior year, if they 
were to be assessed at all. Option two was to mark each core subtest and have the student take only the subtests 
corresponding to the core courses he or she had completed successfully. The second option allowed us to include all 
students but with a considerable amount of missing data. Furthermore, with nine possible subtests, the various 
permutations of subtests each student would be eligible to take was large. 

There were other issues as well. Attempting to ensure that students answered questions only from the 
subtests corresponding to courses they had successfully completed was not entirely successful, as some students 
would complete subtests for courses they had not taken. Having gaps in the ‘Scantron’ answer form produced 
unreliable item analysis results from the electronic printouts. Different sample sizes of students taking different 
subtests required that all item analyses had to be performed by hand. Turning the raw data from the Scantron sheets 
into a usable dataset had to be manually organized and separated by varying numbers of students taking each 
subtest. Thus all point bi-serial calculations had to be done by hand after a considerable amount of organization of 
the data. 


In addition to these somewhat operational data collection and analysis issues, we also encountered the more 
fundamental and impactful issue of faculty commitment. Ideally, at the end of each year of testing each department 
would review the results of their subtests to identify strengths and weaknesses in their respective courses. It was 
suggested that they select a couple of weaknesses and make adjustments in curriculum and delivery in order to 
strengthen these areas. This of course resulted in a significant investment of faculty time and effort, which as Bond 
argues is the main deterrent to the creation and deployment of common examinations (Bond, 2007). While Bond 
notes that the very process of 'developing and coming to consensus on assessment' through thoughtful discussion 
and reflection among those teaching a common core is valuable and productive, our experience was that most of our 
faculty simply did not feel the time invested would produce a commensurate payback. 
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Two factors influenced the decision to discontinue the customized exam and initiate administration of the 
ETS Major Field Test (MFT) (ETS, 2012). The first factor was, frankly, the challenges involved in continuing to 
update and refine both the content and the delivery of the customized exam. This primarily was a faculty issue, in 
that 1) although expertise exists in-house, faculty are not necessarily experts in test design and validation, and 2) as 
is to be expected, faculty naturally tend to prioritize their time by placing more importance on personal teaching and 
research. In that regard, the on-going maintenance of the customized exam was exceptionally time-consumptive, 
and the result, as noted above, was an exam that was becoming increasingly inconsistent in quality across the 
various disciplines. 

Therefore, we initially began to look for a solution that would be simpler to implement and thus more 
sustainable over a longer period of time. We reviewed various options and ultimately settled on the MFT. The 
MFT provided a number of advantages, ranging from the substantive (i.e., questions formulated and validated by 
experts) to the operational (i.e., the related benefit of not having to write, monitor and maintain the questions 
ourselves). 

Transitioning to the MFT altered not only the assessment instrument, but also the timing, venue and nature 
of the target audience. The MFT is directed toward seniors near the end of their program of study, rather than 
toward sophomores/juniors. The MFT also could be administered within a single capstone course that all business 
students are required to take just prior to graduation, meaning that the appropriate venue for administration was less 
ambiguous. With regard to administration of the MFT in the capstone course, we followed the guidance of 
assessment experts, who suggest that not only should time for the exam be provided within a capstone course, but 
also that the students should be held accountable in some way. Katherine Martell, for example, has suggested 
allocating one class period and 5% of the course grade to the MFT (Martell, 2003). This approach has worked well. 

COMPARING THE CUSTOMIZED AND COMMERCIAL APPROACHES 

While the decision to move from a customized to a commercial exam approach was driven primarily by 
operational considerations, it is important to note the relative strengths and weaknesses of the two approaches, and 
to highlight the contributions that each approach can make to an overall assessment program. Toward this end, we 
propose a framework comprised of the various criteria defining the impact of an assessment exam. They include a) 
relevance to learning goals, b) exam design and development, c) exam delivery, d) assessment of learning, e) impact 
on curriculum, and f) impact on student monitoring and management. The framework comparing the two 
approaches is presented in table form, as shown in Table 2. 

SUMMARY AND CONCLUSIONS 

In this paper we have described an evolutionary process that began with a detailed analysis of our mission 
and learning goals, led to the design and implementation of a customized assessment instrument tailored to the 
learning goals, and concluded with a reconsideration of the customized approach and the eventual implementation of 
the standardized Major Field Test. Given the wide array of decision criteria involved in choosing an assessment 
instrument, as summarized in Table 2, a key conclusion arising from this process is that the choice of an instrument 
targeted to program-level goals depends significantly on the individual circumstances of each institution. That is, 
the relative weightings of each criterion will vary. In our case, while each criterion was an important consideration, 
the decision to move to the MFT was influenced primarily by two of them: 1) the benchmarking capabilities of a 
national standardized test, and 2) by our relatively-lesser level of expertise in designing assessment instruments. 

The story does not end there, however, because elements of our customized assessment instrument continue 
to co-exist with the standardized instrument. Lessons learned from creating customized questions tailored to 
learning goals in specific courses have provided input to another important aspect of program-level assessment; i.e., 
the use of common final exams in business core courses. The process used to generate questions for the CCPE, the 
knowledge of assessment attained via that process, and even some of the questions themselves (as well as many new 
ones) are now used in our business school across disciplines to formulate sets of common questions for many of the 
core courses. Like the CCPE, the common questions are targeted specifically to each course’s learning goals, which 
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in turn are linked to the program goals of the undergraduate program. Thus, while it no longer exists as a self- 
contained assessment instrument, the CCPE and the process used to develop it continue to serve an assessment role 
as a complement to our other methods, including the MFT. 


Table 2: Comparison of Customized and Commercial Exams by Criteria 



Customized Exam (CCPE) 

Commercial Exam (MFT) 

Relevance to 

learning 

goals 

• The CCPE is a direct test of the institution’s program 
learning goals (i.e., related directly to what is being taught). 

• Each question on the CCPE is linked to a goal (or goals) of 
the corresponding undergraduate major (i.e., accounting, 
finance, marketing), thus insuring relevance of each question. 

• However - Certain questions are idiosyncratic to the teaching 
approaches and philosophies in the business school. As such, 
questions address what is taught, but not what should be 
taught. 

• The MFT assesses a broader but 
perhaps different set of learning goals. 

• It is a benchmarking tool of what is 
taught across institutions, and as such 
is a potential indicator of appropriate 
and inappropriate learning goals. 

• However - It does not directly address 
the institution’s goals, thus perhaps 
missing aspects of the curriculum that 
provide a competitive advantage. 

Exam design 
and 

development 

process 

• While (or because) the exam is customizable, it also is labor- 
intensive and not necessarily relevant to a faculty member’s 
core interests or skill set. 

• However there are faculty gains in taking the time to engage 
thoughtfully in the process of developing an in-house exam 
thus driving pedagogic changes. 

• Because the exam is essentially pre¬ 
fabricated, little or no additional work 
is required to develop it. 

• MFT questions are developed and 
validated by assessment experts. 

• However this can remove most 
faculty from careful reflection of 
current teaching practices necessary 
for continual improvement. 

Exam 
delivery: 
timing and 
venue 

• The exam can be written to target students at any point in 
their program of study, and covering any particular subset of 
courses. At our institution, the CCPE targeted juniors, who 
had completed their first two years of the business core 
curriculum. 

• However - Prior to the senior year, there is no single course 
that ‘makes sense’ as a venue for a comprehensive 
assessment exam. Therefore : 

o It is much harder to justify to students the relevance of 
the exam in their course (particularly when a grade is 
involved) 

o No sophomore- or junior-level course can assure that 
every student will have taken all of the courses covered 
by the exam. Thus, many students did not. 

• The exam is specifically targeted to 
seniors, and typically is delivered 
within the capstone course. 

• At our institution, because every other 
business course is a prerequisite to the 
capstone, students taking the MFT are 
almost assured of having taken the set 
of courses covered in the MFT. 

• Because the capstone course covers 
essentially all areas of business, it is 
much easier to justify assigning a 
grade for the MFT. 

Impact on 
learning 

• Because it can be delivered prior to the senior year, the 
customized exam can be used as a reinforcement and 
remediation mechanism (i.e., a learning instrument as well as 
an assessment instrument). An individual student performing 
poorly in a particular area of the exam can be provided with 
remedial help. 

• The MFT is an assessment instrument 
presented at the end of a program of 
study. As such, it is not intended as a 
learning or remediation tool. 

Impact on 
courses and 
curriculum 

• Results from the CCPE were used to identify issues in course 
design and delivery, and to implement changes. 

• Results from the MFT were used to 
identify issues in course design and 
delivery, and to implement changes. 

Impact on 
student 
monitoring 
and 

management 

• In addition to the performance results provided, the CCPE 
also provided information that helped guide student 
management. This included identifying and responding to 
students who were out-of-sequence, and to students who had 
not taken all or even most of the courses covered in the 
CCPE. Much but not all of this was due to occasionally lax 
enforcement of prerequisites, which has been addressed and 
significantly improved. 

• Although not pursued, an assessment exam prior to the senior 
year can also be used as a hurdle, which students must pass in 
order to proceed in their program of study. 

• Its position at the end of the program 
of study limits the MFT as a 
mechanism for student management. 
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